A user usually launches a software application by manipulating a mouse, joystick, wheel, game pad, track ball, or other user input device to select the application from a list of files or a group of graphic icons that represent applications installed on a personal computer (PC). Alternatively, the user may enter the path and executable file name for the application in a run dialog box to execute an application.
Another form of user input for executing an application employs touch-sensitive displays that are responsive to the touch of a user's finger or a stylus on the display screen. Touch responsive displays can be pressure activated, responsive to electrical capacitance, changes in magnetic field intensity, or responsive to other variables to determine the location of a finger or stylus contacting the display screen. Another type of touch sensitive display includes a plurality of optical sensors spaced-apart around a periphery of the display screen so that the location of a finger or stylus touching the screen can be detected. Using one of these touch sensitive displays, a user can touch a graphic icon or file name of an application to select it and then tap once (or twice) on the icon or file name to execute it on the PC. However, touch sensitive systems are generally incapable of detecting more than a single point of contact and are typically unable to detect the shape of an object that is proximate to or touching a display screen.
Another approach previously developed in the prior art that might be employed for launching an application uses cameras mounted to the side and above a horizontal display screen to visually capture an image of a user's finger or other object that is touching the display screen. This multiple camera mounting configuration is clearly not a compact system that most people would want to use in a residential setting. In addition, the accuracy of this imaging system in responding to an object that is on or proximate to the display surface depends upon the rather limited capability of the software used with the imaging system to visually recognize objects and their location in three-dimensional (3D) space.
To address many of the problems inherent in other types of touch-sensitive displays and imaging sensing systems, particularly in regard to launching a specific application, a user interface platform was developed in the MIT Media Lab, as reported by Brygg Ullmer and Hiroshi Ishii in “The metaDESK: Models and Prototypes for Tangible User Interfaces,” Proceedings of UIST 10/1997:14-17. This article describes how the metaDESK includes a near-horizontal graphical surface that is used to display two-dimensional (2D) geographical information. Above the graphical surface is disposed an arm-mounted flat-panel display that serves as an “active lens” for use in displaying 3D geographical information. A computer vision system inside the desk unit (i.e., below the graphical surface) includes infrared (IR) lamps, an IR camera, a video camera, a video projector, and mirrors. The mirrors reflect the graphical image projected by the projector onto the underside of the graphical display surface. The IR camera can detect a distinctive pattern provided on the undersurface of passive objects called “phicons” that are placed on the graphical surface. Magnetic-field position sensors and electrical-contact sensors are also included in the metaDESK. The reference describes, for example, how the IR camera detects the IR pattern (which is transparent to visible light) applied to the bottom of a “Great Dome phicon” and responds by displaying a map of the MIT campus on the graphical surface, with the actual location of the Great Dome in the map positioned where the Great Dome phicon is located. Moving the Great Dome phicon over the graphical surface manipulates the displayed map by rotating or translating the map in correspondence to the movement of the phicon by a user.
A similar approach to sensing objects on a display surface is disclosed in several papers published by Jun Rekimoto of Sony Computer Science Laboratory, Inc., in collaboration with others. These papers briefly describe a “HoloWall” and a “HoloTable,” both of which use IR light to detect objects that are proximate to or in contact with a display panel on which a rear-projected image is visible. The rear-projection panel, which is vertical in the HoloWall and horizontal in the HoloTable, is semi-opaque and diffusive, so that objects reflecting IR light back through the panel become more clearly visible to an IR camera as they approach and then contact the panel. The objects thus detected can be a user's fingers or hand, or other objects. In a paper entitled, “CyberCode: Designing Augmented Reality Environments with Visual Tags, Designing Augmented Reality Environments,” DARE (2000), Jun Rekimoto et al. disclose that the IR camera can recognize IR identification (ID) tags attached to objects that are placed on such a device. The reference teaches that “when a device with a camera recognizes these IDs, a predefined action—such as opening a specific web page—launching an application, or starting a movie, is activated automatically.”
While using an ID tag on an object to launch a related application is thus known in the art, there does not seem to be any teaching about responding to a plurality of objects of about simply recognizing objects per se, such as by their shape of other characteristics. For example, it would be desirable to enable an optical sensing system to detect a specific object based on its shape, size, or other physical characteristics and respond by launching a related application. Thus, if a user places a camera on a display surface that includes an optical sensing system, it would be desirable to automatically detect the camera based upon its physical characteristic(s), and automatically launch an application to download images from the camera and to organize the images for storage, or to edit them.
The prior art does not suggest how to deal with the situation that can occur when the application that should be launched in response to detecting an object is not installed on a local storage accessed by the interactive display that detects the object. It would be desirable to automatically respond by downloading the application program from a remote server or other remote storage site, if the application is not installed locally. Once downloaded, the application should be installed, registered to the current user of the interactive display (if appropriate), and then executed.
Also, in some cases, a plurality of applications might be associated with one object, so that it would not be clear which software application to automatically launch when the object is detected on the display surface. Accordingly, it would be desirable to detect a plurality of objects that constitute a plurality of objects uniquely associated with a specific application, and then launch that specific application in response to detecting the plurality of objects or proximate to the display screen.